[UPDATE PRIMITIVE] Fix bugs 1–5 and implement design improvements 1–3, 5 from v2.25.1-next.2 evaluation#199
[UPDATE PRIMITIVE] Fix bugs 1–5 and implement design improvements 1–3, 5 from v2.25.1-next.2 evaluation#199
Conversation
Bug 1: Improve bqrs_interpret -t parameter docs, add database param Bug 2: Fix query_results_cache_compare to derive resultCount from SARIF Bug 3: Include category field in annotation_search FTS matching Bug 4: Add findingId as primary lookup for audit_add_notes Bug 5: Change bqrs_info files param to single file string Design 2: Serialize concurrent database_analyze on same database Design 3: Already implemented (cacheKey/queryName in cache_lookup) Design 5: Auto-enable annotation tools in VSIX with new setting Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/0d52c27d-21be-49ea-bf90-c8fb6f50a3da Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
Add cacheDatabaseAnalyzeResults() to auto-cache SARIF output from database_analyze into the query results cache. Update README with new enableAnnotationTools setting documentation. Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/0d52c27d-21be-49ea-bf90-c8fb6f50a3da Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
Dependency ReviewThe following issues were found:
Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. License Issuespackage-lock.json
OpenSSF Scorecard
Scanned Files
|
There was a problem hiding this comment.
Pull request overview
This PR updates several MCP tools and supporting infrastructure to address multiple bugs found during evaluation (BQRS tooling, audit/annotation behavior, query results caching, and database_analyze concurrency/caching), and adds a VS Code setting to enable annotation/audit/cache tools by default.
Changes:
- Fix and extend CodeQL tool schemas/behavior (
bqrs_infopositional arg,bqrs_interpretsource-context support, audit notes byfindingId). - Improve annotation search semantics (category matching) and query-results cache metadata (persist/derive SARIF result counts).
- Add
database_analyzeresult caching + per-database serialization to avoid CodeQL DB lock contention; update VS Code extension env defaults + tests/docs.
Reviewed changes
Copilot reviewed 15 out of 17 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| server/src/lib/cli-tool-registry.ts | Adds per-database mutex, bqrs_interpret option mapping, and post-database_analyze caching hook. |
| server/src/lib/result-processor.ts | Computes/stores SARIF resultCount; introduces cacheDatabaseAnalyzeResults(). |
| server/src/lib/sqlite-store.ts | Extends annotation search to also match category. |
| server/src/tools/audit-tools.ts | Adds findingId-based lookup for audit_add_notes, with composite-key fallback. |
| server/src/tools/cache-tools.ts | Improves query_results_cache_compare to derive counts from cached SARIF when missing. |
| server/src/tools/codeql/bqrs-info.ts | Renames positional input from files (array) to file (string). |
| server/src/tools/codeql/bqrs-interpret.ts | Adds optional database parameter and improves -t documentation. |
| server/test/src/lib/sqlite-store.test.ts | Adds coverage for category-based annotation search. |
| server/test/src/tools/audit-tools.test.ts | Adds schema + handler coverage for findingId and optional composite-key fields. |
| server/test/src/tools/cache-tools.test.ts | Adds coverage for deriving counts from cached SARIF content. |
| server/test/src/tools/codeql/bqrs-info.test.ts | Updates test for file positional input. |
| extensions/vscode/src/bridge/environment-builder.ts | Auto-sets ENABLE_ANNOTATION_TOOLS and MONITORING_STORAGE_LOCATION based on new setting. |
| extensions/vscode/test/bridge/environment-builder.test.ts | Adds tests for default enablement + override behavior. |
| extensions/vscode/package.json | Adds codeql-mcp.enableAnnotationTools configuration setting. |
| extensions/vscode/README.md | Documents the new VS Code setting. |
| server/dist/codeql-development-mcp-server.js | Updates built artifact to reflect source changes. |
Comments suppressed due to low confidence (1)
server/src/lib/result-processor.ts:363
cacheDatabaseAnalyzeResults()computesresultCountas...length ?? null, which can storenulleven when SARIF JSON is valid (e.g., results omitted). Prefer...length ?? 0after a successful parse so cached entries reliably have a numeric count for SARIF outputs.
// Compute result count from SARIF content
let resultCount: number | null = null;
try {
const sarif = JSON.parse(resultContent);
resultCount = (sarif?.runs?.[0]?.results as unknown[] | undefined)?.length ?? null;
} catch { /* non-SARIF content */ }
|
@copilot Use TDD methodology to:
|
- Fix databaseLocks memory leak: clean up map entries on release - Fix bqrs_interpret source-archive: prefer src.zip over src/ - Fix cacheDatabaseAnalyzeResults: use resolved database path - Fix annotation category search: add COLLATE NOCASE - Fix resultCount: use ?? 0 instead of ?? null (two locations) - Fix VS Code tests: wrap mock mutations in try/finally - Fix codeql_bqrs_info integration tests: files → file in test runner, monitoring runner, and test-config.json Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/52936c89-89e6-4b1d-8c0e-12dcb9ad2388 Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
Applied all PR review comments and fixed the failing integration tests in commit 7854829: PR review fixes:
Integration test fix:
|
Prep for the v2.25.1-next.2 release as a hotfix for the v2.25.1-next.1 release.
|
Resolves #198 |
Both issues fixed with TDD methodology across commits 7a590b4, 1f75bdc, and 886b3f8: 🔴 🟠 Also created |
v2.25.1-next.2 prereleaseThere was a problem hiding this comment.
Pull request overview
Copilot reviewed 45 out of 48 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
server/src/lib/cli-tool-registry.ts:421
- Similarly, auto-resolving
source-location-prefixfromcodeql-database.ymlwill overwrite a user-provided--source-location-prefixif one was passed. It would be safer to only set this value when it’s currently unset, or to treat an explicit value as higher precedence than the derived one.
data-douser
left a comment
There was a problem hiding this comment.
Round 3 Validation Results (March 31, 2026)
I performed a full round of ql-mcp server validation against this build using 2 CodeQL JavaScript databases and the advanced-security/codeql-sap-js custom query packs. All tool categories were exercised.
Bugs 1–5: All Confirmed Fixed ✅
| Bug | Tool | Fix Verified |
|---|---|---|
| Bug 1 | bqrs_interpret |
file accepts string, -t accepts ["id=...", "kind=..."] array, produced valid SARIF (5 results). Also tested with database param — works and resolves source archive. |
| Bug 2 | query_results_cache_compare |
totalResultCount: 5 now matches actual SARIF results across both databases |
| Bug 3 | annotation_search |
Searching "xyzzy-unique-category" (only present in category field, not content/entityKey) correctly returns the annotation |
| Bug 4 | audit_add_notes |
findingId alone works; response says "Updated notes for finding id N" — no need for sourceLocation+line |
| Bug 5 | bqrs_info |
file (singular string) parameter works correctly |
Design Items Verified
- Design 5 (auto-enable tools):
ENABLE_ANNOTATION_TOOLSandMONITORING_STORAGE_LOCATIONconfirmed working in extension config
Design 3: query_results_cache_lookup — Needs TDD for cacheKey/queryName Params
The PR description states "Design 3 ... Already implemented — cacheKey and queryName params exist", but live testing shows this is not working as expected:
mcp_ql-mcp_query_results_cache_lookup(language="javascript")
→ {"cached":false,"language":"javascript"}
This returns {cached: false} even after two successful query_run calls that cached results with keys a61176a8cc05da6c and 098548ec4fc6a959. The tool still only accepts the language parameter — there's no way to look up by cacheKey or queryName.
Requested: Implement Design 3 using TDD methodology:
-
Write failing tests first in
server/test/src/tools/cache-tools.test.ts:- Test:
cache_lookupwithcacheKeyparam returns{cached: true, ...metadata}when matching cache entry exists - Test:
cache_lookupwithcacheKeyparam returns{cached: false}when no match - Test:
cache_lookupwithqueryNameparam returns all cached entries for that query name across databases - Test:
cache_lookupwithqueryName+languagereturns filtered results - Test:
cache_lookupwith onlylanguagestill works (backward compatibility)
- Test:
-
Update the
cache_lookuptool schema inserver/src/tools/cache-tools.ts:- Add
cacheKey: z.string().optional()— direct lookup by cache key - Add
queryName: z.string().optional()— lookup by query name across all databases
- Add
-
Implement the handler logic to query the cache store by key or name
-
Verify all tests pass, including the existing
language-only behavior
New Observation: bqrs_interpret Result Count Discrepancy
bqrs_interpret produces different result counts depending on whether database is provided:
- Without database: 5 results
- With database: 4 results
query_runSARIF: 5 resultsbqrs_inforaw rows: 52
The WITH vs WITHOUT difference appears to be caused by source archive-aware location dedup. Consider documenting this behavior or investigating whether the with-database count should match query_run output.
Other Notes
audit_store_findingsdedup logic works correctly — detected 1 duplicate when re-storing an existing finding- 16 annotations and 4 audit findings were tracked throughout the validation session
profile_codeql_query_from_logsworks correctly for post-hoc profiling
…limit; fix env-builder to not overwrite pre-existing vars TDD tests for Design 3 (query_results_cache_lookup): - Exact cacheKey lookup returns metadata including resultCount - Non-existent cacheKey returns cached:false - Language-only filter correctly returns matching entries - databasePath filter returns matching entries - limit parameter restricts returned entries Environment builder fix (addresses PR review): - Only set ENABLE_ANNOTATION_TOOLS when not already defined in env - Only set MONITORING_STORAGE_LOCATION when not already defined in env - Added test: additionalEnv MONITORING_STORAGE_LOCATION override is preserved Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/97593c7f-7fd2-48f4-9f4c-0e97dfa47902 Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
Addressed in commit 2cfe94b: Design 3 —
The Environment builder: Applied the reviewer's suggestion — |
|
@copilot apply changes based on the comments in this thread |
…log clarity - BQRS file param: check `file !== undefined` instead of truthiness, validate with `trim()` to catch whitespace-only strings - bqrs_interpret database: always delete key from options regardless of value; fail fast on empty/whitespace-only string with clear error - environment-builder: check `process.env` for inherited values instead of the freshly-constructed `env` object (which is always empty) - result-processor: log "result count unknown" instead of "0 results" when resultCount is null - Added tests: empty string file, whitespace file, empty database param, process.env ENABLE_ANNOTATION_TOOLS preservation Agent-Logs-Url: https://github.com/advanced-security/codeql-development-mcp-server/sessions/f9deedbc-dbc0-4432-9d7b-775d1a9f624f Co-authored-by: data-douser <70299490+data-douser@users.noreply.github.com>
All four review comments addressed in commit 24ba5d2:
|
📝 Update Information
Primitive Details
bqrs_interpret,bqrs_info,annotation_search,audit_add_notes,query_results_cache_compare,query_results_cache_lookup,database_analyze, VSIX extension env configUpdate Metadata
bqrs_infoparameter renamedfiles→file(array → string);audit_add_notesparametersowner/repo/sourceLocation/linenow optional whenfindingIdis provided🎯 Changes Description
During a thorough evaluation of the ql-mcp server against 3 JS databases, 4 query packs, and 1,101 findings, 5 bugs and several design gaps were identified. This PR addresses all 5 bugs and 3 of the 5 design improvements, plus two additional issues discovered during follow-up evaluation.
Bug Fixes
Bug 1 —
bqrs_interpretunusable through MCP interface:databaseoptional parameter, mapped to--source-archivefor SARIF source contextsrc.zipoversrc/directory (matching typical CodeQL database layout); throws clear error if neither exists--source-location-prefixfromcodeql-database.ymlmetadata whendatabaseparameter is provided; does not override caller-supplied--source-archiveor--source-location-prefixdatabasekey is always deleted from options regardless of value to prevent it from being forwarded as an unsupported CLI flag; empty/whitespace-only strings fail fast with a clear errorfileparameter: checksfile !== undefined(not truthiness) so empty strings are caught by validation; handles JSON-encoded array strings ('["/path/to/file.bqrs"]') and actual arrays passed by MCP clients; validates withtrim()to also catch whitespace-only strings-tparameter description with explicit KEY=VALUE format documentationBug 2 —
query_results_cache_comparereportstotalResultCount: 0:resultCountwas never populated when caching fromprocessQueryRunResultsruns[0].results.lengthat cache time (using?? 0to distinguish empty results from unknown counts); compare tool falls back to parsing cached content whenresultCountis null, gated on SARIF format to avoid unnecessary JSON parsing for graphtext/csvBug 3 —
annotation_searchignorescategoryfield:(id IN (SELECT rowid FROM annotations_fts WHERE MATCH $search) OR category = $search_cat COLLATE NOCASE)COLLATE NOCASEfor case-insensitive behavior consistent with the FTS unicode61 tokenizerBug 4 —
audit_add_notesignoresfindingId:findingIdas preferred lookup (direct annotation ID) with explicit!= nullcheckowner/repo/sourceLocation/line) optional fallbackBug 5 —
bqrs_infofilesarray causes CLI error:files: z.array(z.string())→file: z.string()to matchcodeql bqrs infowhich accepts exactly one filefile(string) instead offiles(array)Design Improvements
Design 1 —
database_analyzepopulates query results cache:cacheDatabaseAnalyzeResults()— after successfuldatabase_analyzewith SARIF output, stores results in the cache socache_compare/cache_retrievework without re-running queries individuallyresolveDatabasePath()for canonical database paths in cache keys to avoid divergence from positional argsresultCountis null, avoiding misleading operational logsDesign 2 — Serialize concurrent
database_analyzeon same database:resolve()for consistent path representationDesign 3 —
query_results_cache_lookupverified with TDD:cacheKey(exact lookup viagetCacheEntryMeta),queryName,language,databasePath, andlimitparameters all work correctly inquery_results_cache_lookupDesign 5 — Auto-enable annotation/audit/cache tools in VSIX:
codeql-mcp.enableAnnotationToolssetting (default:true)ENABLE_ANNOTATION_TOOLSandMONITORING_STORAGE_LOCATIONenv varsprocess.envfor inherited values from the extension host process to avoid overwriting pre-existing env values (the freshly-constructedenvobject is always empty, soin envchecks would always pass)additionalEnvstill overrides for advanced users🔄 Before vs. After Comparison
API Changes
🧪 Testing & Validation
Test Results
cacheKey/queryName/language/databasePath/limit), 9 bqrs-interpret definition tests, 2 bqrs-interpret handler behavior tests, 2 file parameter coercion regression tests, 1 SARIF result count derivation test, 1 environment builderMONITORING_STORAGE_LOCATIONpreservation test, 1 environment builderENABLE_ANNOTATION_TOOLSprocess.envpreservation test, 1 empty string file parameter test, 1 whitespace-only file parameter test, 1 empty string database parameter testcodeql_bqrs_info/basic_infoandcodeql_bqrs_info/json_formatintegration tests by updating test runners and fixtures to usefile(string) parameter📋 Implementation Details
Files Modified
server/src/tools/codeql/bqrs-interpret.ts— addeddatabaseparam with updated description, improved-tdocsserver/src/tools/codeql/bqrs-info.ts—files→fileserver/src/lib/sqlite-store.ts— category match in FTS search withCOLLATE NOCASEserver/src/tools/audit-tools.ts—findingIdprimary lookup with explicit!= nullcheckserver/src/tools/cache-tools.ts— SARIF content fallback for result count, gated on SARIF formatserver/src/lib/result-processor.ts—resultCountcomputation with?? 0+cacheDatabaseAnalyzeResults()with clear log messaging ("result count unknown" when null)server/src/lib/cli-tool-registry.ts— per-database mutex with cleanup,bqrs_interpretdatabase→source-archive+source-location-prefix mapping (always deletesdatabasekey, fails fast on empty string, preferringsrc.zip, guarding against overwrite of explicit options), file parameter defensive coercion with!== undefinedcheck andtrim()validation, analyze cache call with resolved DB path, alphabetically sortedfsimportsextensions/vscode/package.json—enableAnnotationToolssettingextensions/vscode/src/bridge/environment-builder.ts— auto-set annotation env vars checkingprocess.envfor inherited values from the extension host processextensions/vscode/README.md— document new settingCHANGELOG.md— corrected version reference (v2.25.1-next.1 → v2.25.1-next.2)server/test/src/tools/codeql/bqrs-interpret.test.ts— 9 definition tests (schema validation, Zod array rejection), 2 handler behavior testsserver/test/src/lib/cli-tool-registry.test.ts— 4 handler behavior tests (source-location-prefix auto-resolution, file param defensive coercion), 3 empty/whitespace edge case tests (empty string file, whitespace-only file, empty database param)server/test/src/lib/sqlite-store.test.ts— category search testserver/test/src/tools/audit-tools.test.ts— findingId testsserver/test/src/tools/cache-tools.test.ts— SARIF count derivation test, 5 new TDD tests for cache lookup (cacheKey/queryName/language/databasePath/limit)server/test/src/tools/codeql/bqrs-info.test.ts— updated forfileparamextensions/vscode/test/bridge/environment-builder.test.ts— annotation setting tests withtry/finallycleanup,MONITORING_STORAGE_LOCATIONprocess.envpreservation test,ENABLE_ANNOTATION_TOOLSprocess.envpreservation testclient/src/lib/integration-test-runner.js—files→fileforcodeql_bqrs_infoclient/src/lib/monitoring-integration-test-runner.js—bqrs→fileforcodeql_bqrs_infoclient/integration-tests/primitives/tools/codeql_bqrs_info/json_format/test-config.json—filesarray →filestringDependencies
🚀 Compatibility & Migration
Backward Compatibility
Breaking Changes
codeql_bqrs_info:files(array) →file(string). Callers passingfiles: ["/path/to/file.bqrs"]must change tofile: "/path/to/file.bqrs".audit_add_notes: Previously requiredowner+repo+sourceLocation+line. Now acceptsfindingIdas alternative. Existing calls with composite key still work unchanged.API Evolution
bqrs_interpret.database,audit_add_notes.findingIdbqrs_info.files)🔗 References
Related Issues/PRs